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ABSTRACT 

The unbiased estimate of a "treatment effect" reached 
by analysis of covariance in a nonrandomized experiment would often 
require that a different covariate be used in each treatment. A 
sufficient but unlikely condition for an unbiased estimate is that 
the covariate for each treatment is (1) the complete covariate that 
predicts the outcome as fully as possible from initial 
characteristics of the case, or (2) the complete discriminant that 
fully represents differences between group means on the initial 
characteristics. When the covariate for a treatment is a weighted 
composite of (1) and (2), the adjusted outcome mean may have positive 
or negative basis, the bias being a nonlinear function of the 
weights. If (1) and (2) do not wholly account for the covariate, the 
adjustment to the outcome mean is reduced in absolute magnitude. A 
procedure is offered to correct for error of measurement when there 
are two or more fallible covariables. . Specif ication errors as well as 
errors of measurement have an attenuating effect, however. Since the 
parameters of the specification errors sre unknown, no correction 
procedures can be counted on to provide an unbiased estimate of the 
treatment effect. (Author/BW) 
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Analysis of covariance — Angel of salvation, or temptress and deluder? 
Lee J. Cronbach, David R. Rogosa, Robert E. Floden and Gary G. Price 
Stanford Evaluation Consortium, Stanford University 

In comparing social and educational programs, treatment effects 
are confounded with effects of subject, characteristics when self- 
selection or a nonrandom assignment rule determines who enters (or 
who completes) a treatment. 

In an attempt to remove confounding, the investigator might con- 
fine his comparison to subsets of cases whose distributions on one 
or more initial measures are the same within all treatments. It 
was established long ago (Rulon, 1941; Thorndike, 1942) that such 
post hoc matching of cases produces a biased conclusion. Analysis 
of covariance (ancova), the usual current method of coping with non- 
equivalence, uses the within-groups regression of outcome on a char 
acteristic measured prior to treatment to adjust the outcome measure. 
This purports to estimate the treatment difference for persons who 
were alike at the outset. It is in effect a matching procedure; 
but few users of ancova have been aware that it too is biased. 

Two decades ago Cochran (1953, esp. p. 689) drew attention 
to the bias, but he was almost unique among statisticians in making 
this point. If the covariate does not fully reflect the difference 
between experimental and control groups, he suggested, analysis 
of cc ance underadjusts. More forcefully, Campbell and Erlebacher 
(1970) charged that statistical bias may have produced the negative 
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conclusion in the West: ngh oust evaluation of Head Start. They asserted that 
whenever the treatment groups differ initially, ancova will not adjust ade- 
quately for the initial group differences. The adjusted outcome mean of 
the "more aul?" group will be unduly high, relative to the adjusted out- 
come mean of tho "less able 11 group. Much the same argument appears in 
Campbell and Boruch (1975). 

Their assertion is rapidly becoming part of the conventional wisdom 
of social scientists, A recent authoritative statement on policy research 
(Riecken and Boruch, 1974, p. 109) not only asserts that ancova and 
similar methods "underadjust for the latent group differences 1 ' but 
recommends that the person analyzing posttests "should live with such 
pretest differences rather than try to adjust them away ..." As other 
examples of this pessimism in social science, we note Lord's statement 
(1967, p. 305; see also Lord, 1969) " . . that with the data usually 
available for such studies there simply is no logical or statistical 
procedure that can be counted on to make proper allowances for uncontrolled 
preexisting differences between groups," and of Meehl f s statement (1970, 
p. 402) that a comparison made in an ex post facto experiment 
is "in most instances so radically defective in its logical structure 
that it is in principle incapable of answering the kinds of theoretical 
questions which typically give rise to its use." 

When cases are not assigned at random tc treatments, the samples 
receiving the treatments are best thought of as coming from separate 
populations (Thorndike, 1942). These multi-population studies. are 
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Analysis of covariance — Angel of salvation, or temptress and deluder? 

Lee J. Cronbach, David R. Rogosa, Robert E. Floden and Gary G. Price 
Stanford Evaluation Consortium, Stanford Univei ity* 

Abstract 

The estimate of a "treatment effect" reached by analysis of 
covariance in a nonrandomized experiment depends on the covariate chosen. 
An unbiased estimate of the treatment effect would often require that a diffe 
covariate be used in each treatment. A sufficient but unlikely condition 
for an unbiased estimate is that the covariate for eac! -eatment is (a) the 
complete covariate that predicts the outcome in that treatment as fully as 
possible from initial characteristics of the case, or (b) the complete 
discriminant that fully represents differences between the group means 
on the initial characteristics. When the covariate for a treatment is a 
weighted composite of (a) and (b), the adjusted outcome mean for a treat- 
ment may have positive or negative basis, the bias being a nonlinear 
function of the weights. To the extent that (a) and (b) do not wholly 
account for the covariate, the adjustment to the outcome mean is reduced 
in absolute magnitude. A procedure is offered to correct for error of 
measurement when there are two or more fallible covariables. Specification 
errors as well as errors of measurement have an attenuating effect, however. 
Since the parameters of the specification errors are unknown, no correction 
procedures can be counted on to provide an unbiased estimate of the 
treatment effect. 
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called quasiexperiments in the social-science literature, observational 
studies in the statistical literature. For a recent review of the statis- 
tical literature on such studies, see McKinlay (1975). Because of selection 
or self-selection, the populations have different distributions of initial 
characteristics. 

The most direct problem of inference is to describe the relation of 
outcome to initial characteristics in one population at a time. As Cronbach 
and Furby (1970, p. 78) suggested, the wi thin-treatment regression equation 
relating outcome to true initial status 1 on relevant covariates describes 
the effect in a suitable way. This inference, made separately in each popu- 
lation, seems to present no insuperable difficulties. It is the comparison 
of treatments that encounters serious difficulties. 

This paper concentrates on the estimate of the intercept a within 

A 

Population A. The bias in the estimate is described as a function of the 
underlying parameters of a linear model for quasiexperiments. Once the 
bias arising from a particular covariate is characterized within each treat- 
ment, it is a simple matter to characterize the bias in the estimated dif- 
ference a A - a g , arising from applying that covariate in each treatment 
as in ancova. It is also possible to evaluate $ A and a g each with a differ- 
ent (optimal) covariate; the bias in this difference is another function of 
the same parameters. The conditions affecting bias also affect statistical 
efficiency, not necessarily in the same way; we do not pursue questions of 
efficiency here. 



True score" is an ambiguous concept. For present purposes the reader 
can think in terms of classical test theory (Lord & Novick, 1968), or of 
theory for congeric tests (Joreskog, 1971). 
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As Cochran and Rubin (19 73) implied, ancova does not always 
underadjust. The conditions under which the treatment given the 
less able group will be underrated or overrated can be specified; 
soire of these conditions are indicated by Weisberg and Bryk (1974). 

We shall show that an adequate adjustment or evaluation of bias 
requires information that is unlikely to be obtainable. 

Our thinking grows out of exchanges with other investigators, and 
is influenced by their published papers and drafts and by their reactions 
to preliminary versions of our own thinking. Jn particular, 
we have received help from conversations with Robert Boruch, Donald T. 
Campbell, Albert Erlebacher, and their associates, with David E. Wiley 
and Ward Keesling, and with Janet D. Elashoff , Arthur S. Goldberger, 
Frederic M. Lord, and Donald B. Rubin. We have benefited also from inter- 
changes with James W. Bush and his group studying the methodological 
problems as seen in research on delivery of health services. Rosedith 
Sitgreaves and other colleagues in the Technical Seminar of the Stanford 
Evaluation Consortium have made helpful comments. We hope that responses 
to this preliminary report will extend our list of creditors. 

Definitions, notation and model 
Assume that an investigation is intended to appraise .ne effects of 
Treatments A and B on outcome Y. To consider here more than two treat- 
ments, more than one outcome, etc., would obscvre what is rsovel in our 
argument. To appraise the treatmer" is *:o determine the expected value 
of Y in that treatment, for cases whose initial characteristics are known. 
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Notation and s ampling as sumptions 

We speak of Populations A and B, and of their union Population G, 
the grand population. This two-population model reduces to the one- 
population model, the true experiment, when corresponding parameters of 
the two populations are equal. 

Population membership reflects eligibility rules, readiness to 
volunteer, etc. In any subgroup of G having uniform initial 
characteristics, one can specify the probability of belonging to 
Population A conditional on initial characteristics , and its complement 
the probability of belonging to B. Integrating these over the density 
distribution of initial characteristics in Population G, we arrive at 
unconditional probabilities: A member of Population G fall.: into Popula- 
tion A with probability tt^. It is convenient to write K for 
^A^B = 7T A^*~ 7T A^" Wnen an after-school science program attracts only 
a select group of students, tt a is perhaps 0.05 and k is 0.05/0.95=0.053. 

As Weisberg and Bryk emphasized, variables may correlate differently 
in Populations A, B, and G. Score composites that are uncorrelated in 
A (or B, or G) may be correlated in the other two populations. 
Geometric intuitions about dimensions, angular separations, etc. cannot 
be trusted, unless one is careful to specify and keep in mind one 
particular population. 

Formally, it is assumed here that cases have been drawn at random 
from Population A to receive Treatment A, etc. Cases may usually be 
thought of as persons. An investigator who applies a treatment 
to groups (e.g., to schools, classes, or communities) should take this 
larger aggregation as the case, unless he is prepared to assume that 
persons or other units within an aggregation are treated independently. 
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Every case p is described at the outset of the experiment by a 

vector of true scores £ . (i = 1, 2, • • . ) • A may be combined with 

pi i 

an error variable e to generate a X^. Values of are random, not 

fixed. W will represent any variable (any X or £ or combination of 

them) ciiosen as a covariate. To distinguish variables we write W-^ and 

or £. and £^ , etc. will refer to the true score corresponding to W- , 

1 J W]_ 1 

etc. 

We use subscripts where necessary to identify the distribution con- 
sidered. E.g., y x ^ and X ^(A) are the means of X i in Population A 
and in a sample from Population A, respectively. The code is as follows: 

(A), (B) Calculated within Treatments A and B, respectively. 

(T) Calculated within Treatment T, where T may be A or B . 

(A,B) Calculated from the pooled distribution of scores 

expressed as deviatio ; from the respective treat- 
ment means. Population A,B is an artificial, but 
statistically valid, population. In Sample A,B 
the numbers of cases in A must be k times the num- 
ber in B or else weighting must be used. 

(G) Calculated on the grand population or on a pooled 

sample with A and B cases in the ratio, k:1. 

We make an important scaling assumption, without loss of generality: 



for any g ± , V Q) - 0. Also, y £(G) = 0, hence p = 0 . 

Consequently, y x ^ = -ku x 



l i 



For any e and T, it is assumed that u /rrN = 0. This denies that fal- 

e t U 

lible information (X^ as distinct from the corresponding influences 

which treatment a person gets. (But see Note 4, p. 17). It will not be 

assumed that , in general, a (A) = a or a = a ( . 

i i i J i j 

Indeed, all homogeneity assumptions will be avoided. 



10 



or observable in practice) are included in the set of any Combination 

of the 5 l is itself a The set is to be thought of as indefinitely 

large. An investigator would have exhaustive information on case p if 

he had a set of scores for p on all the £ . 

i 

A formal restriction is needed if the matrix of p_ is n ot 0 f 

full rank. Components of the E >± , orthogonal within T, can be identified, 
and any ^ can be expressed as a weighted sum of these components, the 
weights possibly differing with the treatment. If any linear combination 
of £j has a vector of weights in T that are proportional to the weights 
of 5^, we assume it to have the same absolute coefficient of variation 
(!a/y|) as £> ± . (If the ratio of the vectors is negative then the coef- 
ficients of variation are opposite in sign.) This requirement i s imposed 
in Population A and Population B, separately. Although hypothe ical cases 
can be constructed in which the assumption is violated, it appears impos- 
sible for a violation to arise in practice. 

5 is defined as the £ that best accounts for the pooled Vithin- 
treatments variance, and £^ is defined as the £ ± that best predicts Y 
within Population T. When the chosen covariate W is the best predictor 
of Y in Population A,B — that is, when £ is chosen as the covariate — 
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The investigator cannot be sure of the direction of bias in his 
Even if (hypothetically) he knows that his covariate lies in fi , the 
investigator who selects a single arbitrary has no knoWleo.ee of 



2 

the residual variance 0 Y -w(A B) is minimized. To complete the defini- 

tions: Py£*(A B) - ° and P Y£*(T) - °" Unless 5 and are the saTne ' 

P Y£*(T) > P Y£*(T)" 111 shouid bc noted that P Y?*(T) can be ne S ative in 

one treatment. Pr*r*/ T v ma y ^ e negative also (for example, if one treat- 

ment capitalizes on extroversion and one on introversion). 

2 

Arbitrarily we designate the treatment for which P^*(t) > ^ as 
Treatment A. Defining A on the basis of 5 makes A M the more able group, 
insofar as that label can be given a priori meaning. The investigator 
who has less than exhaustive inforxation on his sample is not able to iden- 
tify £*, hence he cannot directly identify which treatment is A. 

He will presumably think of the group whose sample mean on W is 
higher as M the more able group 11 , if the sign of OyW(T) is P ositive in 
both treatments. Where P W (x) does hav£ the saTne si£n in both treatments » 
we polarize W to make the sign positive. Where tae Q^CT) are °PP osite 
in sign, we polarize W to make the sign positive in A. The treatment for 
which U W ( T ) > 0 is designated T w - 
Model accounting for Y 

C is the error observing Y. U is a disturbance taking into 
account any unpredictable event that affects the true score on Y; it 
includes variations in the delivery of the treatment, and events such as 
illness that could not be predicted from initial status. ^ (x) = ^(x) = ®* 

We assume u T> and C linearly independent. For case p in 

Population T, 

<» Y Tp = 0t T + B Y^(T) ^p +U Tp + S • 



2 If y_ = 0, the assignment of the label A is a matter of in- 
difference. ^ Tnfere will be other statements where at a boundary a descrip- 
tion or polarization is equivocal. We do not comment on other special 
cases of this kind. ^ ^ 
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Having set ^y(G) at zero ' we can identify a T as the expected 
value of Y for cases in Population T whose £* p = p * . ^en 

P S*S*(A) = °C*£*(B) = 1 and e Y?*(A) = S Y£*(B)' a A * "b des "ib^ 
the treatment effect at every value of £,* . 

Interest in the a is diminished when the regressions of Y on 
any t >± are heterogeneous (as will occur when any of the three conditions 
just stated is violated). That parameter is emphasized in this paper, 
however, because it has been the focus in previous uses of ancova. Even 
with heterogeneous regressions, a T is relevant for decisions where one 
proposes to apply the same treatment to all cases in the future. Suppose 
an investigator comparing Textbooks A and B believes that Pr*r* = 1, 

6 Y£*(A) >S Y£*(B)' and a A > V A di s°rdinal interaction of this kind 
might suggest assigning students above a certain level of to A, and 
the others to B. The investigator may, however, recommend uniform adop- 
tion of Textbook A hereafter. Such a policy would be reasonable whenever 
the costs of keeping two treatments in place outweigh the benefits from 
differentiated assignment. 
The complete discriminant £° 

In general, Populations A and B will be differentiated in several 
respects. We introduce the construct of a "complete discriminant" (which 
will coincide with the linear discriminant function under restrictive 
assumptions). Consider a set of orthogonal components Z^ Z 2 . . . 

derived from the matrix of p„ , , „ . 

h% (G) 

The complete discriminant £° is defined so that in either T, for all 
k. I Pairs, p ZkC o (G) : = : p ^ (T) . r° is polarized 

to make = _^ 0(b) > 0 . 
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Group membership depends on the characteristics of the case that 
influence the decision to enter and remain in a treatment and/or the char- 
acteristics considered in assignment to treatments. (An error e can func- 
tion as a discriminant; e.g., a student who by chance earns a higher SAT 
score than his true score may as a consequence apply to and be accepted 
by a more selective college. To make our argument as direct as possible 
we have ruled this out by specifying ^(t) = 0> hence P^o e (c) = ® ^ or 
all e. But see Note 4, p. 17.) 
Adj usted outcome 

By "adjusted outcome 11 we mean Y - By^^W. If ^ is the chosen 
covariate, we have the ideally adjusted outcome: 

(2> V ? Tp = Y Tp- B Y^(T)^p = \ + % + S • 

Since the expected values of v and £ over persons within T are 
zero, the mean of this adjusted outcome within Sample T is an 
unbiased estimate of a^. The central question in this papr~ has to do 
with the bias in ot^ when the chosen covariate W is not 

Traditional formulations and procedures in the light of this Model 

The constructs represented by £^ and £° do not appear explicitly 
in the literature on ancova. The usual model is expressed in terms of a 
single covariate, more or less as follows: 



(3) 



Tp 



Homogeneity of regressions for the particular W — ^YW(A) = ^YW(B) = 
f^YWCA B ) — is explicitly assumed. To apply ancova, an investigator 
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collects data on one or more X^^ that he considers relevant. If he 
observes more than one X he forms the composite W that maximizes 

2 n u • 

r YW(A,B) " O the ^wise, his arbitrarily chosen X i becomes W. 

Using a sample estimate of 6 VW(A ^ B) , the investigator can estimate 
a A and a B ° r he can estimat:e a A - a g in one step. He usually tests 
the null hypothesis a A - = 0, assuming that values of W are 

fixed, liking W as random is appropriate in a two-population 
experiment; but statistical theory for random covariates is limited. 

The traditional assumption of homogeneous regressions can be 
weakened by adding a term in U T Tp x T to (3), T being a dummy 
variable. The analysis proceeds much as before (Atiqullah, 1964; 

Cramer, 1972; Goldberger, 1972b), ,. n e treatment effect being described 
as the difference between the adjusted treatment means evaluated at 
the grand mean on X . 

Since the 1950s, statisticians have been conscious that error. of 

measurement makes the usual estimate of a - a inadequate. L?rd 

A B 

(1960) developed a procedure for use in a two-population experiment 

when two fallible, equivalent measures W ad W 1 have been made, 

the corresponding r being fixed. Lord estimates what a - a 

w A B 

woi:ld be if £ were known and chosen as 
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the covariate. Lord's formulation and the work of Porter (1967) and 
DeGracie (1968) seem implicitly to assume that one variable is the best 
predictor of Y in Populations A, B, and A,B, and that this variable 
has been chosen as the covariate. Unpublished notes by various authors 
have touched on the problem of correcting for error of measurement when 
the represent two or more £^ . At best, a solution along the 

lines of those notes must assume that these can be combined into 

a variable £ such that p r * r ( . = p * , . = I . Some theoreticians 
have been aware that an error-free but incomplete covariate adjusts the 
treatment effect inadequately, the most complete discussions being those 
of Cochran and Rubin (1973) and Weisberg and Bryk (1974). 

Although we have not seen any formal use of the construct t, , 
the concept of £ appears in considerations of model I.fication 
(e.g., DeGracie and Fuller, 1972, p. S30) . The literature on structural 
models of social and economic processes refers to a model as being 
completely specified if all causal influences are included. That 
is, model specification is complete if a set of 5^ included in 
the model can be combined to evaluate £^ 

A person who (hypothetically) had ideal information 
in a two-population or a one-population experiment should compute 

B vr */ A N and 3 v r* /T5N . Entering these in (2) and averaging over cases 
A Yc,g(lJ; 

would lead to unbiased estimates of a. and ol,. This procedure makes 

A D 

no assumption of homogeneity, and it does not require using the same 
covariate in each treatment. 
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Notes on heterogeneity of regression 

Before turning to the analysis of bias, we comment on aspects of 
the homogeneity assumption. 

(1) The investigator whose data fail to reject the null hypothesis 
^YW(A) " ^YW(B) = ® cannot confidently conclude that the regressions 
are homogeneous. With samples of the usual size, the test on regression 
slopes is not powerful enough to detect important differences (Cronbach 
and Snow, in press, Chapter 4). Having regarded the regressions onto 
W as homogeneous, the investigator usually interprets - as 
describing the expected effect in Population G for any value of 
other initial characteristics—but this may not be valid even when 
^YW(A) = 6 YW(B)- 

(2) It does not appear likely that the same variable will be the 
best predictor of outcome in both populations, as different treatments 
can be expected to call upon aptitudes in different proportions. Unless 
both the p £*£*(t) are l * the regressions of Y onto not more than one 
linear combination of and are homogeneous. If both Pr*r*/ T ^ =1 ' 
the regressions may or may not be homogeneous. A similar statement can be 
made about observed scores. Hence it is a priori unlikely that whatever W 
the investigator chooses will yield homogeneous regressions. 

(3) Procedures suggested for taking error of measurement into 
account have assumed that W has the same reliability in each treatment. 
This seems not to be an appropriate assumption in a two-population 
experiment, since a* (A) usually will not equal O* Unless one 
does assume equal reliabilities, homogeneous regressions onto W imply 
nonhomogeneous regressions onto 
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&rj. estimated with an arbitrary, wholly relevant covariate 

Assumptions and dimensionalization 

In this section we confine attention to relationships in Population A 
(not the sample), and t:o a covariate that within A is a linear function of 

and The initial argument will be developed under the following 

assumptions : 

1. o<p^, (A) < I . 

2. Dimensions I and II define the plane SI in which and 

A A 

^ lie - p i^(A) = 1 < Pi ika) = 0 • and 

2 

R 0 = I 

K £ "I II (A) 1 ' 



3m y H(A) > ° " ( From Assumption I, Uj^j > 0 .) 
A - °I(A) = a iI(A) = 1 • 

5. The covariate W fi is an arbitrary linear combination of 

A A 

and £ ; i.e., W n lies in Q A . 

A 

A 

Consistent with a polarization rule made earlier, p * > 0 

5 a\ (a) 

A 

We now set out to relate the bias in ot to the following set of 

A 

parameters (all within Population A): y^. , , , y^ + y^ 



A 



and p_* 



. Note that y = y-*/<j-* . Figure 1 displays the variables. 
' ^A A A 

Because I is at the top center of Figure 1 to conform to the scheme of 
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Figure 1. A configuration of variables in plane Q 
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15 

later figures, we measure angles in the clockwise direction (contrary to 

convention). The coordinates of the centroid are {y y ✓ « \ t y ».} # 

I^A) II \A) 

Assumptions 1 and 3 locate the centroid in Quadrant 1 of the plane. We denote 

the angle from £ to £° by A ; 0° < X < 90° . (The earlier assump- 
tion that V r * /A x > 0 will restrict A within this range when Pr*r*,^ 
5 (A) ^B (A) 
« 0 and tt a « 7T fi . ) The angle <t> between W fi and may range 

A 

between -90° and + 90° ; cos $ = p * f . Hereafter 

A 

we omit subscripts (A), as all parameters are defined within Popula- 
tion A. We write 8 A for ^y^*(A)' anc * We sim P^^^3 r ft^ to ft . Tha eovar- 

A 

iances of the variables with Dimensions I and II are given in Table- 1, 



TABLE I 



Covariances of Variables with the Dimensions 



a 2 



< X 0 % 

= a^ 0 cos A = a^ Q sin A 



2 

W Q a cos 4 a sin 4. ° w 

Q Q « 
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Analysis 



(4) W 0 = (cos 40 o„ I + (sin 4.) c TT II 

From ( 1 ) , 



(5) Y - a A + B A O^* I + u A + ? . 

A 

(6) y TT = (cos <f>) a |i + (sin 4>)c p 



and 



(7) = a A + 6 A . 

The regression equation for predicting Y can be written as 



(8) E (Y|w n ) = y Y + (cos +) e A o^* (^ - ^ )/o 

A. u6 Ob 



Except when cos <f> = 1, the regression slope in (8) is smaller than 
the slope in (5) . 

Equation (8) is not biased, as a description of the relation of 
outcome to . Bias enters when a ^ , estimated by entering 
= 0 in (8) , is taken as a measure of . 

Bias as a function of <j) 

The choice of covariate has a critical influence on the estimate 
of the treatment effect. Inserting W ^= 0 into (8), 



(9) a 
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The last terra in (9) is the "adjustment" that would be made to Viy( A ) with 
as the covariate in ancova. According to (7), g A 0 {r *\} T is the ideal 



adjustment that changes Viy(A) 
Substituting from (6), 



to a 



A " 



00) a AW = y Y * (cos <J>) 3 A a ? * ^(cos ^ + (sin <j>) u n J • 

Q A 

The bias = - a A . Because the centroid lies on £ , 

^I^II = cos ^ : s * n ^ ' Us i n S th: "- s fact we can reach 



(II) 



cos A - cos (J) cos(A-<j>) 



"2 + 2 r 



This equation is one of the three main results in this paper. 

A = 0 when <j> = 0 or <j> = A . I.e., if is used as the 

covariate, bias in is eliminated. Also if the complete discriminant 

o ^3 
£ is used as the covariate, bias in a ATT is eliminated. So far as we 

AW 

know, this latter observation has not been made previously.^ 

Figure 2 plots cos <j> a nd as a function of <J> and shows how the 

W J2 



Indeed, if an incomplete discriminant that ^satisfies Pr^u.r 0 ^ = ® 
is used as the covariate, there is zero bias in . A 

In our mathematical development such instances do not appear explicitly; 
they enter as instances where an attenuation effect exactly offsets an 
overestimate. 

4 

Weisberg and Bryk, who formulate the problem differently and have no 
variable in their model that maps into £° , nonetheless arrive at a 
somewhat similar finding. Likewise, Arthur Goldberger (1972a,b and personal 
communication) has established that if one forms the groups by explicitly 
dividing Population G on a certain variable — which then becomes £° — 
taking that variable as a covariate estimates the treatment effect without 
bias. In the literature on educational and psychological field studies, 
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Figure 2. Adjustment to be subtracted from mean outcome 

for Treatment A as a function of <{) , where 

V A) 

the covariate W lies in fl . 
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adjustment and the bias vary. In the region 0° < <j> < A , the bias is 

negative; that is to say, ancova overadjusts and P V(A) and Treatment A is made 

look less effective for persons at the grand mean than it is. In the 

regions -90° < <J» < 0° and A < $ < 90°, the bias makes Treatment A look 

better than it is. If -90° < <f> < A -90°, the estimate of a is greater 

A 6 

(more seriously biased in favor of A) than when no covariance adjustment 

is made; that is, a > jjl . Conclusions about the net bias in a -a n 

(A) A B 

will be derived from tMs information in a later section. 

The shape of Figure 2 changes with X . When X = 0° the 

adjustment is always nonnegative and the adjustment curve is symmetric 

about <{) = 0 . When X = 90° the ideal adjustment is zero and 

the adjustment curve encloses equal areas above and below the zero 

adjustment axis. For any X the maximum of the adjustment occurs 

at <f> = A/2 and is proportional to 1 + cos X . The minimum occurs 

at <J> = (A/2) -90° and is proportional to -1 + cos X . Figure 2 is 

coded at the top to show where > 0 and hence where A is T^ 

Also, it is coded to show where the bias makes T rT look better 

w 

than it should. 



4 (continued) 

explicit assignment to one treatment of persons in a certain range of 
the intended coyariate is known as a "regression discontinuity 11 design. 
Explicit assignment is almost invariably based on a fallible X rather 
than a £ , as Goldberger comments. Goldberger f s proof implicitly 
assumes that any loss of subjects subsequent to assignment depends 
only on £ and/or random events. 

Goldberger has pointed out (personal communication) that our argument 
becomes more inclusive if we allow a fallible measure to enter into the 
complete discriminant. We have ruled this out for ease of exposition. 
So long as all variables lying in ft are free of measurement error, 
any measurement error can be identified with a variable III or IV 
(below). This permits comparison of our results with those of previous 
papers on disattenuation in ancova. It would be possible to restate our 
argument, writing X for the complete discriminant, bearing in mind that 
the mean of the measurement error in Population A is not zero. Then ft 
would be defined by and X°, and various W Q would replace the 

of our argument. (Paradoxically, if W Q is a variable lying in the 1 
plane defined by and the fallible X°, using the true 

covariate would have an attenuating effect because f 



R2 r .r*vo < 1 0 



as 

*ft 
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The investigator cannot be sure of the direction of bias in his a 

AW 

Even if (hypothetical^) he knows that his covariate lies in fi , the 

investigator who selects a single arbitrary W fi has no knowledge of <j> 

beyond the fact that 0 < | cos <f> | < | Pyw J . N or does he Uave knowledge 

of X . With two covariates known to lie in SI , he would oe in an ideal 

position. If Wj and W 2 lie in Q , the multiple-regression equation 

for predicting Y within A yields C , which produces a bias-free a 

A A 

(See pages 27-30.) 

It takes some imagination to construct a plausible substantive 
example of a covariate where -90° < <j> < X - 90° . 

For the sake of example, let Treatment A be training in assettiveness 

and let the dependent variable be some kind of verbal performance. Assume 

that verbal aptitude £ is the best predictor of the dependent variable 

in Populations A and B as well as A,B. For purposes of the example, 

the discriminant £° is a measure of gentleness; persons above a certain 

level of gentleness (opposite of assertiveness) tend to opt Into the 

experimental treatment. Suppose also that on the average gentler persons 

have superior 5 scores. Say, X « 60° . Now let the investigator 

choose as covariate a role-playing pretest in which the person is to 

assert justifiable counter-arguments in response to some communication. 

Assume that this W lies in fi^ . Role playing may well correlate 

substantially but negatively with £° , and have a small positive 

correlation with 5 . Say, (f) = -80° . If so, the 

covariance adjustment would make ^ > p > ^ § ig say> 

the attempt at adjustment worsens the bias. 
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With regard to the bias in Treatment B, one important change 

is required. We can distinguish <(>. and 4>_ , X k and \- 9 fi A and . 

A B A B A B 

(<f> B = cos _1 p ? * (B) , X B = cos^pgjgo^) , B B = 6 Y? * (B) .) 

d U- B B 

D 

(10,) "BW fi = ^Y(B) " (GOS V B B°^ [ (C ° S V ^I(B) + (sin V y H(B)] 

B 



Changes here are in notation only. In (11) , however, the fact that 
^I(B) anc * ^11 CB) are ne 8 ative requires a change cf sign: 

1/2 

= - B B o P * [cos X B - cos*, cos(A B - + M? T/ ^ 



A BW„ = " [ C ° S X B " COS *B C ° S( V V] [^(B) + ^l(B)] 



B 



All preceding statements about the bias in a can be suitably 

A 



modified to appl}*- to a . 

D 
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Alternative assumptions 

We now consider the alternative assumption -1 < p^ D ^ ^ < 0, 

A 

hence Uj^j < 0 . (To keep y IX ( A j > 0 is an arbitrary polarization 
that does not lose generality.) Under this assumption 90° < A < 180°. 
Equations (9) to (11) remain valid. We proceed without subscripts 

(A). 

The figure tracing the adjustment as a function of <f> would be a 
transformation of Figure 2. Since y I < 0, the ideal adjustment is 
negative. If A = 120° instead of 60° as in Figure 2, the new curve 
can be obtained by inverting the mirror image of the old one. The scale 
for the adjustment is inverted, with 8*0*.* Vi T now below zero. There is 
a small hump above zero where A - 90° < <f> < 90°, and a larger half- 
cycle swinging below zero to the left of A - 90°. Again, a is 
unbiased when either g* or £° is used as the covariate. (If <f> is 
plotted over the range -90° to 90°, -£° is located at <f> ■ A - 180°.) 
The discussion on pages 17-19 is valid under the alternative 
assumption considered here, save that certain statements about 
ranges of <{) and A must be reworded. 
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Estimate of ot A -a_ with wholly relevant covariates 

This section uses the results within treatments to 

examine the bias in the difference a^- a fi . In analysis of covariance, 

the same covariate is applied to each treatment. The following discussion 

considers first the bias resulting when a covariate is chosen separately 

in each treatment. This section is limited to wholly relevant covariates; 

2 

that, is, the covariate chosen in A satisfies R,, *.**. n = 1 and 

V-5 a 5°(a) 

2 

the covariate chosen in B satisfies R, T r * rri , v - 1 

Vn*5 b 5°(B) 

Separate covariates 

Figure 2 depicted the bias in with covariate W^ 

A 

Precisely the same function applies to B, with 5? , W n , etc. replacing 

B n fi 

their counterparts in A. In general, $ f (j> and \. f X_ ; 

A B A B 

the ideal adjustment in A equals that in B ouly in special cases. 



The bias in a A - a„ equals ( ot A - ot A ) - ( a_ - a_ ) 

A B * A A 7 B B 7 

or A AW n * V n " hereafter, A A - A fi . If W fi 

is C K or £° £nd w n is £,> or £° , A = A„ - 0 , and consequently 
A & A B 

the estimated difference has zero bias. With certain choices of covariates, 

A A > A_ so that the net bias makes A appear more advantageous then 
A B 

it is. The reverse occurs with other choices of covariates. We proceed to 
examine how the net bias relates to the parameters of our model. 
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Holding X. and A fixed, the net bias (or the net adjustment) 
A B 

can be representad by a function of <J>. and $ . The contour map in 

A D 

Figure 3 is generated with \^ 60° , A fi = 30° , K — 1 , and 

B A°C*tt) = Vs*(B) • SinCe ^*(A) = °- 5 V(A) ^ %W = °- 866 V(B) 

-0.866 u r O/ A v , these parameters cause the ideal adjustment for B 

to be larger ir absolute value than that for A. The curves in Figure 3 

are contours of the net bias, A - A- . Where "Bias favors A" appears, 

A D 

there is a hill representing a positive bias; and where "Bias favors B ff 
appears, there is a depression representing negative bias. If the range 

were extended in any direction the contour map would repeat. (The net 
bias differs from the net adjustment by a change of signs and by addition 
of a constant. Note the curve labelled "zero adjustment".) 

The configuration would be much the same for other values of 



A and A . In general, the bias favors A over most of the joint 
A B 

range of the $ and <j> . As the investigator does not know the 4> T 
A B 

p o 9t , however, it is impossible for him to specify the direction of 
5 S A IA) 

the net bias arising from a particular covariate. 
A single arbitrary covariate 

Ordinarily the user of ancova chooses a single covariate or forms a 

single composite covariate. Suppose that the chosen covariate W is wholly 

2 2 

relevant in- both treatments, i.e., **w.£*£°(a) ™ R W-? A C°(B) = * " ^ S ° an 

A B 

simplify the results of the preceding section by placing a restriction on 
the relation of to ft . 

D A 



and 
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Figure 3. Net bias with a separate, wholly relevant covariate in 
each treatment. - 60° , Xg ■ 30° . The diagonal line represents 

the coordinates for which <j> - X «= <j> - X . 
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One such restriction is to specify ^.\.^o (A) = ^. 5 * ? o (B) = 1 • 

Then can be mapped into Q , and for any W n <j> - A = <f> - A . 

« 06 A B B B 

We align the two planes to make £° a common reference line. In Figure 3 

a straight line identifies all points such that d> — A « <f> - X 

A A B B 

When a single W fi is used in both treatments, the bias indicated at 

the point on the line that corresponds to $ and <J> results. 

A B 

In general, the bias favors A over most of the length cf the line. 
The investigator who tries to pick a relevant covariate will try 
to pick one close to «■* or £° or both; this will locate his 
study in the center of the chart. For the parameters in Figure 3 
the resulting net bias is moderate, but may favor either A or B. 
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Attenuating effect of irrelevant information 
and error of measurement 



An arbitrary covariate 

2 

Now consider a general covariate W such that I n(A) < 1# 

Assumptions 1-4 of page 14 are retained. We define a variable III TT 

w 

2 

that is uncorrelated with I and II within A, and set O ( . = ] 

J-J"L W (A; 

It is convenient to keep P^-y(^) > 0. In this section we again drop 

A 

most subscripts (A). 

The covariate W is a combination of a and a residual 

identified with III^. (Ordinarily, each W is associated with a 

different III^ , and any given W may be associated with one 
in A and another in B .) This residual may represent error of 
measurement, or it may represent true information that neither predicts 
Y within A nor discriminates between groups. In place of (4) 
we have 



(12) W = (cos <{>) a I + (sin <{>)a II + h III_ - W + h HL. 

2 2 

Thus a > a w . Since III W is a residual, P m ^ is zero, 

Q W 

and y TT = M TT is the same as in (6) . 
Now 



(13) = (cos <J>)a w Uj + (sin <j))a w u I3 - 



£3 

We write P^ (A) for rJ. I 11(A) = 0 W fi /a w * It: should be n ° ted that 



2 



p*" T/ . is not a reliability coefficient unxess R_ _ « 1; that is, 
h (Ay Sy 1 

2 2 

unless the true score lies in Q A . We do not assume P W ( A ) = p w(B) * 

2 

From (5) and (12) , = cos <j> &pp£* . Then ,3^ = I = 

2 

(cos ^)^ A a ^* a y / a y » the adjustment term ^yW^W is 

A £2 

2 

(cos (J)) $ A 0£* [(cos (J))y I + (sin <f>) u^] . Comparing this with the adjust- 
A 

ment term in (10) , we see that including irrelevant information in the 

covariate reduces the adjustment. The adjustment to the outcome mean 

2 

in A , given by W , is Py(^) t: i- mes that given by . This 

A 

is our second main result. 

Figure 4 plots the adjustment, tracing the attenuating effect 

2 

for specified values of Py( A ) • Ordinarily, however, when one changes 

2 

the balance of I and II within W , a change in P TT / A \ results, as 

W (A) 

2 

will be seen below. Unless Py(^) * s high* ancova with any W 

will underadjust (a > a ) provided that Pt-*rO/ A \ * s positive. 

A A A 

But when this correlation is negative, the opposite is true (a < a ) . 

The adjustment described above generalizes a result in classical 

2 

lest theory: Tf £ x is the true score corresponding to X, = . This 

X X 

equation describes the attenuating effect of error of measurement on a 
regression coefficient, the multiplier on the right being the reliability 
coefficient. Several waiters, starting with Lord (1960), set out to 
correct the covariance adjustment for error of measurement, retaining 

the assumption ^r\9(A) = ^YW(B) * They a PP lied in various wa y s the 

2 

principle that dividing 3yW(A,B) by p ^ W(A)B) S ives the slo P G of 
pooled wi thin-groups regression of Y on £^ . This is true 



33 



26 

only if wi thin-group reliabilities are equal, which is not necessarily 
the case with nonrandom assignment. 

We now find that irrelevant information (i.e., reflecting 

dimensions other than I and II) has the same effect as measurement 
9 2 

error. Since p* < , dividing 6^ by the reliability 

coefficient is not an adequate correction when irrelevant information 

TT 5 

enters W. 

Bias in ct^ - . To consider the net adjustment a new figure 

similar to Figure 3 could be produced for any specified values of 
2 2 

P W(A) and P W(B) * Figure 3 itself can be reinterpreted when 
2 2 

K = 1 and P W ( A ) = Py(B) • ' nie cur v e labelled "zero adjustment 11 

retains that label. In the region enclosed by that curve the net 

bias is reduced in absolute value. The large positive adjustments 

2 

are reduced, and if P W ( T ) is sufficiently low there is no longer 
a region where bias favors B. This indicates a set of conditions 
making the Campbell-Erlebacher generalization valid (if A , not 
T^ is considered to be the treatment given the "abler group"). 



5 2 
Campbell and Erlebacher suggested using as a divisor. 

A memorandum circulated privately by Campbell in 1972 withdrew the 
proposal, but a similar idea appears with a new rationale in Campbell 
and Boruch (1975). Since P^ T) < P 2 £* W ( T ) < P W(T) » P YW M 

divisor overcorrects. As can be seen by considering the possibility 
that £° is the chosen covariate, such a correction would sometimes 
make matters worse. 
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Two arbitrary pretest measures 

We noted earlier that when two W^ have been observed, it is 

T 

possible to combine them into a covariate proportional to 5^ which 
provides a bias-free adjustment of ^y(X) ' W ^ th measures that lie out- 
side tt^ 9 the relationships are less simple. 

When two or more X. have been observed, the usual procedure in 
analysis of covariance is to form the composite that best predicts Y 
within treatments pooled. To analyze within treatments, an analog would 
be to form the composites W Y (A) anc * W Y(B) t * Mt ^ est P re dict Y within 

the respective treatments. Even if £ were to lie in fim, however, 

W Y(T) 

it would be unlikely that ^ = 1 or that p^ 0 ^ ' (t) = 1 " 

*T W Y (T) W Y(T) 

Thus the true score on whatever combination of observed scores 

best predicts Y gives an a T with an unknown bias. Disattenuation of 

the regression coefficient in T by the reliability coefficient 

2 2 
p £ u or by p would not correct adequately. 

*W Y(T) Y(T) W Y(T) 

The ideal adjustment could be determined if certain quantities were 
known. Assume that Wj and W 2 have been observed, and write III and 
IV to replace the III^ . Again, consider relations within A only. 

(14) W, = (cos <M 0 I + (sin <f>. ) 0 II + h III 

1 W lfi 1 W lfi i 

W, = (cos O a TT I + (sin <J> ) a„ II + h III + g„IV . 

Here = cos -1 p <f> = cos p f * , 

W 1^A ' W 2JPA 

°i = •■■ °iv 2 = 1 and V m = ^iv = °- 
If errors are independent, = 0. But errors of observation 

36 
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can be correlated. (See Cronbach et al., 1972, Chapter 9.) The covariance 
matrix for certain variables of interest takes the form of Table 2, which 
reflects the basic logic of Cronbach et al. on disattenuation when errors 
are correlated . In the covariance matrix we have substituted 
u 1 = (cos <f> )a , v « (sin <f> )a , u « (cos <f> ) a , v « (sin <{> 9 )a TJ 

1 in 1 1 w m 1 1 w 2fi 2 2 w 2« 

and q = UjU 2 + VjV 2 . xhe quantities whose sample values are directly 
observable are underlined in the table. 
We note that 

2 2 

2 U . 2 + V i 2 2 ° W 2n U 2 2 + V 2 2 

PW. 2 ~ 2 2 ' and p y 1 5 2 5 5 

' 0 W, U l + V I + h . " 2 ^ u 2 2 + v 2 2 + h 2 2 + g 2 2 



2 2 

P W ' ' anc * h ] h 2 were known, the investigator could estimate 

Y from 

(15) Y = 6 YW -W W lfi + 6 YW -W W 2fi • 

Call this regression estimate . Since correlates perfectly 

with £^ , the composite serves as a complete covariate. 

(u 2 + v 2 ) ( Ul 6 A a *) - (u 2 6 A a c *)q 

From Table 2, L „ = ■ _ 

* W 2fi 

( 2 j. 2 N t 2 ^ 2 N 2 

(u l V l^ (u 2 V 2^ ~ q 

and a similar formula holds for (3._ 

™2a' W lS! 
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TABLE 2 



Covariance Matrix 



w l 


W 2 




ff 2Q 


Y 


2 2,2 


q + hjh 2 


?. 2. 


q 


U i a Y^(A) /a ^(A) 


W 2 


2 2,2 2 
u 2 + v 2 + h 2 + g 2 


q 


2 2 
u 2 + v 2 


U 2 a Yq ( A) /a ^(A) 






2 2 


q 


U i a Y^(A) /a ^(A) 


W 2fi 






2 2 
u 2 + v 2 


U 2 a YC*(A) /r ^(A) 


Y 








a 2 
Y 
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The ideal adjustment (see p. 17 ) is S ,y * = £ *y a.* . 

^A ^A ^A 



Since 



y in = V IV = ° ' 



(16) 



Thus, 



V f 6 YW - W y W + ^YW - W = W 

W A YW lfi W 2fi 1 2Q in W 2 1 W A 



(17) Mr* = y w * 0 £ */o * . 

^A A ^A W ft 

a * 

Since p yw , = p y? * , = ^ J£ 

A A A A 51 



Then from (17) 3^* y w * = 3 Y £ ; ny v , and the ideal adjustment can be 

A A S\ ^A 

expressed in terms of the parameters in (14). 

This procedure for disattenuation is our third main result. 

If III and IV consist of uncorrelated measurement errors only, the 

procedure is equivalent to that of Keesling and Wiley (1975). They 

assumed h 2 = 0 and interpreted III and IV in this way; in general 

we find it necessary to allow for specification errors and for 

2 2 

correlated errors. p w , p y , and h 1 h 2 are unobservable. If 

reliability coefficients are used and an assumption is made about 
h 2 , a partial disattenuation is achieved. 
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Conclusions and implications 
for one- and two-population experiments 

In the two-population experiment three variables are critical: 
the complete covariates and and the complete discriminant g°. 

In principle, analysis of covariance procedures give a less satisfactory 
description of treatment effects than determination of the regression equation 
m within each treatment separately. Ideally one could regress the outcome 
in Treatment T onto , getting an unbiased estimate of , 

the expected outcome averaged over all persons in the grand population. 
Op is also unbiased when £ is used as a predictor. 

Calculation of a single adjustment in the usual manner is warranted only 
under much stronger homogeneity assumptions than are usually recognized. 

An arbitrarily chosen covariate will in general lead to a biased 
estimate of (or a A - a R ) . Bias also results from the usual 

practice of combining two or more predictors with the weights that best 
predict the outcome within treatments pooled, and using the composite as 
a covariate. The adjustment for initial differences made in analysis 
of covaijance may overadjust or it may underadjust. With some configura- 
tions of data the covariance adjustment gives an estimate more biased 
than the unadjusted difference in outcome means. This is ,t underadjustment ,, 
carried to an extreme. 

As has often been noted, error in measuring the covariate reduces 
the absolute magnitude of the adjustments for A and B. 
Disattenuation on the basis of a reliability coefficient is inadequate 
to remove the bias. Two kinds of specification error also produce 
bias. When the covariate W is perfectly predicted by 5* and £° 
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together (R^ ^o^ ~ but not by either of them alone, the bias 

^ . -1 

is a sinusoidal function of cos o * . When the true score £ rT contains 

5 T W(T) ^ w 

a component that is not perfectly predicted by and/or £°, that 

specification error has the same adjustment-reducing effect as an error 

of measurement. 

In the one-population ("true") experiment, adjustment is needed only 
to overcome differences between the A and B samples . The argument of 
this paper regarding the bLss resulting from using a covariate other than 
£^ or £° could be restated to apply to sample data in the one- 
population experiment. £° would be replaced by an X° (see footnote 4 , 
p. 17), and a wave function similar to Figure 2 would be found. The ideal 
adjustment will ordinarily be smaller than the ideal adjustment in a 
two-population experiment, since £(<p) » arising only from sampling 
error, is likely to be s^iall and therefore the 5* are also small. 

Disattenuation in the one-population study changes the slope of 
the regression line, but does not change the estimated intercept. 
This distinctive aspect of the one-population experiment has not 
been explicit in previous warnings about the consequences of using 
a fallible covariate. 

The investigator working with actual data from a two-population 
experiment does not know how his chosen covariate relates to and g°. 
Therefore, in interpreting an adjusted treatment mean or a within- 
treatmpnt regression equation , he cannot assert that he has taken 
initial characteristics of the treatment population fully into account. 
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Nor can he state that whatever treatment difference he finds, after 
adjustment by analysis of covariance with or without correcting for 
unreliability, is biased in a known direction. 

The sufficient conditions for an accurate description of the 
treatment effect are these: The investigator has observed at least 
two initial characteristics of the cases, and V 9 and an out- 

come Y. He also knows the proportion of variance in W and in W, 

1 4 

not predicted by a linear combination of the complete covariate and 
complete discriminant. And he knows the covariance between the un- 
predicted portions of the two measures. 

These conditions wili not be met in practice. Perhaps it will 
be possible to set pi ausible upper and lower bounds on the 

unobservable parameters, and from them to derive bounds on the 

estimated treatment effect or, better, on the regression equation 

relating outcome to the ideal covariate. An alternative is simply 

to describe the function relating outcome within a treatment to a 

particular set of covariates. (5 , £ , and the true score 

W Y(A) W Y(B) 

on the sample discriminant function would be a good choice.) Such 
a function for Treatment A could be interpreted as describing 
relations in Population A, with no implication that the result 
generalizes to Populations B and G. If functions for the two 
treatments are compared, it must be realized that r acification 
errors have not been allowed for. 
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conditions. Moreover, what is planned as a one-population experiment is 
subject in some degree to the same difficulties of interpretation as a 
two-population experiment, once subjects in the initial sample 
withdraw or fail to provide data. Even the one-population experiment 
is subject to serious problems of external validity (Rubin, 1974; 
Cronbach, 1975), so that the inferences made from it are not 
necessarily more dependable than those from a two-population 
experiment covering a wider range of conditions. 

Our final position is not so pessimistic as those of Riecken 
and Boruch, Lord, and Meehl. We do favor pinning down results within 
populations separately rather than trying to adjust initial differences 
away. Regression onto covariates is needed in that process, 
and the fact tha: covariates are incomplete or imprecise 

does not argue against their use. Comparison of populations is 
certain to be wanted, and the problems encountered in ancova are no 
more than one instance of the ubiquitous problem in social science of 
incomplete specification. The solution is not to abandon realistic 
social science but to make less presumptions claims regarding the 
result . 



43 



35 



BIBLIOGRAPHY 



Atiqullah, M. The robustness of the covariance analysis of a one-way 
classification. Biometrika , 1964, 5J_, 365-372. 

Campbell, D. T. & Boruch, R. F. Making the case for randomized assignment 
to treatments by considering the alternatives: Six ways in which 
quasi-experimental evaluations in compensatory education tend to 
underestimate effects. In C. A. Bennett and A. A. Lumsdaine (Eds.) 
Evaluation and experiment . New York: Academic Press, 1975, 195-285. 

Campbell, D. T. & Erlebacher, A. How regression artifacts in quasi- 
experimental evaluations can mistakenly make compensatory 
education look harmful. In J. Hellmuth (Ed.), Compensatory 
education: A national debate . Vol. III. The disadvantaged child . 
New York: Brunner/Mazel, 1970, 185-210. 

Cochran, W. G. Matching in analytical studies. American Journal of 
Public Health , 1953, Part I, 43 (6), 684-691. 

Cochran, W. G. & Rubin, D. B. Controlling bias in observational studies: 
A review. Sankhya-B , 1973, 35 , 417-446. 

Cramer, E. M. Significance tests and tests of models in multiple 
regression. American Statistician , 1972, 2j> (4), 26-30. 

Cronbach, L. J. & Furby, L. How we should measure "change" — or should 
we? Psychological Bulletin , 1970, J74, 68-80. 




36 



Cronbach, L. J.; Gleser, G. C. ; Nanda, H. & Rajaratnam, N- The 

dependability of behavioral measurements ; Theory of o ^ alizability 
for scores and profiles . New York: Wiley, 1972. 

Cronbach, L. J. & Snow, R. E. Aptitudes and instructional methods . New 
York: Irvington Publishers, in press. 

DeGracie, J. Analysis of covariance when the concomitant variable is 
measured with error. Unpublished doctoral dissertation, Iowa 
State University, 1968. 

DeGracie, J.S., and Fuller, W.A. Estimation of the slope and 
analysis of covariance when the concomitant variable is 
measured with error. Journal of the American Statistical 
Association , 1972, 6_7 , 930-937. 

Goldberger, A. S. Selection bias in evaluating treatment effects: 
Some formal illustrations. University of Wisconsin, Institute 
for Research on Poverty: Discussion Paper 123-72, April 1972 (a). 

Goldberger, A. S. Selection bias in evaluating treatment effects: The 
case of interaction. University of Wisconsin, Institute for 
Research on Poverty: Discussion Paper 129-72, June 1972 (b). 

Joreskog, K. G. Statistical analysis of sets of congeneric tests. 
Psychometrika , 1971, 36, 109-133. 

Keesling, J. W. & Wiley, D. E. Measurement error and ^he analysis of 

quasi-experimental data. Unpublished Draft Manuscript, UCLA, 1975. 



Lord, F. M. Large-scale covariance analysis when the control variable 

is fallible. Journal of the American Statistical Associati on, 1960, 
55, 307-321. 

Lord, F. M. A paradox In the interpretation of group comparisons. 
Psychological Bulletin , 1967, 68, 304-305. 

Lord, F. M. Statistical adjustments when comparing preexisting groups. 
Psychological Bulletin , 1969, 72., 336-337. 

McKinlay, S. M. The design and analysis of the observational study — 

a review. Journal of the American Statistical Association , 1975, 70, 
503-520. 

Meehl, P. E. Nuisance variables and the ex post facto design. 

In Radner & S. Winokur (Eds.), Analysis of theories and methods 
of physics and psychology, Vol. IV. Minnesota Studies in the 
Philosophy of Science . Minneapolis: University of Minnesota Press, 
1970, 373-402. 

Porter, A. C. The effects of using fallible variables in the 

analysis of covariance. Ph.D. dissertation, University of 
Wisconsin, 1967. 

Riecken, H. W. & Boruch, R. F. (Eds.) Social experimentation , New York: 
Academic Press, 1974. 

l?ubin, D. B. Estimating causal effects of treatments in randomized 
and nonrandomized studies. Journal of Educational Psychology , 
1974, 66, 688-701. 

46 



Rulon, P. J. Problems of regression. Harvard Educational Review , 1941, 
11, 213-223. 

Thorndike, R. L. Regression fallacies in the matched groups experiment. 
Psychometrika , 1942, 2 ( 2 ), 85-102. 

Weisberg, H. I. & Bryk, A. S. On the use of the analysis of covariance 
with nonrandom assignment of experimental units to treatment groups. 
Unpublished Draft Report [Huron Institute, Cambridge, Mass.H, 1974. 



47 



